Gemma/[Gemma_1]Using_with_Ollama.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tce3stUlHN0L"
      },
      "source": [
        "##### Copyright 2024 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "tuOe1ymfHZPu"
      },
      "outputs": [],
      "source": [
        "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dfsDR_omdNea"
      },
      "source": [
        "# Gemma - Run with Ollama\n",
        "\n",
        "This notebook demonstrates how you can run inference on a Gemma model using  [Ollama](https://ollama.com/). Ollama is an easy-to-use solution for running LLMs locally and provides built-in support Gemma.\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/gemma-cookbook/blob/main/Gemma/[Gemma_1]Using_with_Ollama.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MwMiP7jDdAL1"
      },
      "source": [
        "## Setup\n",
        "\n",
        "### Select the Colab runtime\n",
        "To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model. In this case, you can use a T4 GPU:\n",
        "\n",
        "1. In the upper-right of the Colab window, select **▾ (Additional connection options)**.\n",
        "2. Select **Change runtime type**.\n",
        "3. Under **Hardware accelerator**, select **T4 GPU**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gJaQ-OVoPKCo"
      },
      "source": [
        "## Installation\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4VYdPAwS9sCr"
      },
      "source": [
        "Install Ollama through the offical installation script."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DHrOMaOAPSAM"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            ">>> Downloading ollama...\n",
            "############################################################################################# 100.0%\n",
            ">>> Installing ollama to /usr/local/bin...\n",
            ">>> Adding ollama user to video group...\n",
            ">>> Adding current user to ollama group...\n",
            ">>> Creating ollama systemd service...\n",
            "WARNING: Unable to detect NVIDIA/AMD GPU. Install lspci or lshw to automatically detect and install GPU dependencies.\n",
            ">>> The Ollama API is now available at 127.0.0.1:11434.\n",
            ">>> Install complete. Run \"ollama\" from the command line.\n"
          ]
        }
      ],
      "source": [
        "!curl -fsSL https://ollama.com/install.sh | sh"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wn2lC5hVPUxy"
      },
      "source": [
        "## Start Ollama\n",
        "\n",
        "Start Ollama in background using nohup."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "q8o4A6QKPp3d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "nohup: redirecting stderr to stdout\n"
          ]
        }
      ],
      "source": [
        "!nohup ollama serve > ollama.log &"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "76mRtotdPu9N"
      },
      "source": [
        "## Inference\n",
        "\n",
        "Run inference using command line."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "k8y8SD1XPzAr"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The capital of France is **Paris**.\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!ollama run gemma:7b \"What is the capital of France?\" 2> ollama.log"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MMg2unvIOtH4"
      },
      "source": [
        "Generate a response via REST endpoint"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_JihVtFsOwsn"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:56.14899689Z\",\"response\":\"The\",\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:56.178231303Z\",\"response\":\" capital\",\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:56.207532308Z\",\"response\":\" of\",\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:56.236605028Z\",\"response\":\" Portugal\",\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:56.265333563Z\",\"response\":\" is\",\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:56.294147887Z\",\"response\":\" **\",\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:56.323264861Z\",\"response\":\"Lis\",\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:56.35282411Z\",\"response\":\"bon\",\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:56.382855843Z\",\"response\":\"**.\",\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:56.413118162Z\",\"response\":\"\\n\\n\",\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:56.619448677Z\",\"response\":\"\",\"done\":true,\"done_reason\":\"stop\",\"context\":[968,2997,235298,559,235298,15508,235313,1645,108,1841,603,573,6037,576,21539,181537,615,235298,559,235298,15508,235313,108,235322,2997,235298,559,235298,15508,235313,2516,108,651,6037,576,21539,603,5231,49147,3839,168428,109,235322,615,235298,559,235298,15508,235313,108],\"total_duration\":518226208,\"load_duration\":1258094,\"prompt_eval_count\":18,\"prompt_eval_duration\":45308000,\"eval_count\":17,\"eval_duration\":470410000}\n"
          ]
        }
      ],
      "source": [
        "!curl http://localhost:11434/api/generate -d '{ \\\n",
        "  \"model\": \"gemma:7b\", \\\n",
        "  \"prompt\":\"What is the capital of Portugal?\" \\\n",
        "}'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FlLNziE93xdP"
      },
      "source": [
        "Chat with Gemma via REST endpoint"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AW-ex-Fgs_La"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:56.80317128Z\",\"message\":{\"role\":\"assistant\",\"content\":\"The\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:56.832244294Z\",\"message\":{\"role\":\"assistant\",\"content\":\" capital\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:56.864473026Z\",\"message\":{\"role\":\"assistant\",\"content\":\" of\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:56.894548916Z\",\"message\":{\"role\":\"assistant\",\"content\":\" Spain\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:56.924834821Z\",\"message\":{\"role\":\"assistant\",\"content\":\" is\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:56.954322472Z\",\"message\":{\"role\":\"assistant\",\"content\":\" **\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:56.984517118Z\",\"message\":{\"role\":\"assistant\",\"content\":\"Madrid\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.014076809Z\",\"message\":{\"role\":\"assistant\",\"content\":\"**.\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.044390316Z\",\"message\":{\"role\":\"assistant\",\"content\":\"\\n\\n\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.073914656Z\",\"message\":{\"role\":\"assistant\",\"content\":\"Madrid\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.104846737Z\",\"message\":{\"role\":\"assistant\",\"content\":\" is\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.134785148Z\",\"message\":{\"role\":\"assistant\",\"content\":\" a\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.166072957Z\",\"message\":{\"role\":\"assistant\",\"content\":\" city\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.195707466Z\",\"message\":{\"role\":\"assistant\",\"content\":\" in\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.224466411Z\",\"message\":{\"role\":\"assistant\",\"content\":\" central\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.252835434Z\",\"message\":{\"role\":\"assistant\",\"content\":\" Spain\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.282191186Z\",\"message\":{\"role\":\"assistant\",\"content\":\" and\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.31179531Z\",\"message\":{\"role\":\"assistant\",\"content\":\" has\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.341159754Z\",\"message\":{\"role\":\"assistant\",\"content\":\" been\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.370877089Z\",\"message\":{\"role\":\"assistant\",\"content\":\" the\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.400127634Z\",\"message\":{\"role\":\"assistant\",\"content\":\" capital\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.430518407Z\",\"message\":{\"role\":\"assistant\",\"content\":\" since\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.459726063Z\",\"message\":{\"role\":\"assistant\",\"content\":\" the\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.489232296Z\",\"message\":{\"role\":\"assistant\",\"content\":\" Middle\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.521220399Z\",\"message\":{\"role\":\"assistant\",\"content\":\" Ages\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.550727167Z\",\"message\":{\"role\":\"assistant\",\"content\":\".\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.579056428Z\",\"message\":{\"role\":\"assistant\",\"content\":\" It\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.611030309Z\",\"message\":{\"role\":\"assistant\",\"content\":\" is\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.648551096Z\",\"message\":{\"role\":\"assistant\",\"content\":\" known\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.686423573Z\",\"message\":{\"role\":\"assistant\",\"content\":\" for\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.71824871Z\",\"message\":{\"role\":\"assistant\",\"content\":\" its\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.74884782Z\",\"message\":{\"role\":\"assistant\",\"content\":\" rich\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.778573494Z\",\"message\":{\"role\":\"assistant\",\"content\":\" history\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.808624406Z\",\"message\":{\"role\":\"assistant\",\"content\":\",\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.838451581Z\",\"message\":{\"role\":\"assistant\",\"content\":\" culture\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.867754055Z\",\"message\":{\"role\":\"assistant\",\"content\":\",\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.896675551Z\",\"message\":{\"role\":\"assistant\",\"content\":\" and\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.925461064Z\",\"message\":{\"role\":\"assistant\",\"content\":\" vibrant\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.956364599Z\",\"message\":{\"role\":\"assistant\",\"content\":\" nightlife\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:57.985969319Z\",\"message\":{\"role\":\"assistant\",\"content\":\".\"},\"done\":false}\n",
            "{\"model\":\"gemma:7b\",\"created_at\":\"2024-07-08T10:53:58.015675807Z\",\"message\":{\"role\":\"assistant\",\"content\":\"\"},\"done_reason\":\"stop\",\"done\":true,\"total_duration\":1298987591,\"load_duration\":1110482,\"prompt_eval_count\":23,\"prompt_eval_duration\":43108000,\"eval_count\":41,\"eval_duration\":1212429000}\n"
          ]
        }
      ],
      "source": [
        "!curl http://localhost:11434/api/chat -d '{ \\\n",
        "  \"model\": \"gemma:7b\", \\\n",
        "  \"messages\": [ \\\n",
        "    { \"role\": \"user\", \"content\": \"what is the capital of Spain?\" } \\\n",
        "  ] \\\n",
        "}'"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "[Gemma_1]Using_with_Ollama.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}